Forecasting Crowd Work Quality via Multi-dimensional Features of Workers

Modeling changes in individual crowd worker performance over time offers new ways to improve the quality of crowd labels, such as by dynamically routing label annotation tasks to workers more likely to produce reliable labels. Whereas prior crowd annotator models have typically adopted a single generative approach, we formulate a discriminative, flexible feature-based model. This allows us to combine multiple generative models and integrate additional behavioral evidence, enabling better adaptation to temporal variance in worker accuracy. Experiments with a public crowdsourcing data show that our model improves prediction accuracy by 26-36% across workers, enabling 29-47% improved quality of crowd labels to be collected at 17-45% lower cost. Furthermore, we confirm that our proposed model shows significantly accurate prediction than baselines under limited supervision.